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Abstract. We presents the results of an extensive numerical study of the phase diagram 
of the spin-1/2, J1-J2-J3 Heisenberg model on a square lattice, for both ferromagnetic and 
antiferromagnetic nearest-neighbor interactions Ji, using exact diagonalization with periodic 
and twisted boundary conditions. Comparison is made with published spin wave calculations. 
We show that quantum fluctuations play a very important role, changing both the extent and 
the wave vector of classical spiral phases, and leading to new quantum phases where the classical 
spiral states have a high degeneracy. These include a new phase with small or vanishing spin- 
stiffness, in addition to known valence- bond- solid and bond-nematic phases. 



1. Introduction 

The effects of quantum fluctuations on the ground-state is a major question in frustrated 
magnetism. One of the most studied examples is the J1-J2 Heisenberg model with spin S = 1/2 
on a square lattice with competing antiferromagnetic (AF) first and second neighbor exchange, 
where the nature of the non-magnetic ground-state that occurs around the maximally frustrated 
point J2/J1 ~ 0.5 has been much debated. Suggestions include valence bond crystals (VBC's) 
with columnar, plaquette, or mixed columnar-plaquette order, and spin liquid (SL) states. One 
approach to this problem is to include a third- neighbor exchange J3, and a recent study of the 
J1-J2-J3 model by Mambrini et al. using exact diagonalization (ED) in full and reduced basis 
sets found plaquette VBC order extending down to the J3 = limit pQ. 

In past years, several classes of layered vanadate and cuprate compounds, which are well 
approximated by a spin-1/2 Heisenberg model on a square lattice have come to the fore 0EJH]. 
Some of these seem to be well described by a J1-J2 model with AF J2, and either FM, or AF Ji. 
However others have more complex properties which hint at longer range interactions. This 
motivates us to consider J1-J2-J3 models with both FM and AF J\ 

In this proceedings paper, we report results of further studies of the J1-J2-J3 model: 

n = Ji £ s i ■ s j + J 2 Si ' s i + Js Yl Si • s j W 



that include now both the case of FM and AF J\. We set J\ = — 1 (Ji = 1) when J\ is 
FM (AF) throughout. As before we principaly used exact diagonalizations (ED) on clusters 
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Figure 1. (Color online). Groundstate phase diagram for the spin-1/2, J1-J2-J3 Heisenberg 
model on a square lattice for J\ = — 1. Symbols indicate the ground state found in 
exact diagonalization of a 32-site cluster, allowing for twisted boundary conditions : FM - 
ferromagnetic; S(q,q) or S(q, 0) — spiral; SR S(q,0) — sort-range order (see text); C(ir/2, 0), 
C(7r/2, 7r/2) or C(tt,0) — collinear; d-BN — quadrupolar (bond-nematic) state; SG — spin- 
gapped. Dashed lines are the classical (S = 00) phase boundaries. 



of N = 16,20,32,36 spins with periodic boundary conditions (PBC), and N = 16,20,32 spins 
with twisted boundary conditions (TBC), complemented by spin wave analysis following previous 
calculations of Chubukov [7 J and Rastelli et al. [HE]. 

We find a wide variety of ground states, including both collinear and spiral phases, and states 
with no conventional magnetic order. For the case of FM J\ our present phase diagram is a 
refinement of that proposed in [6]. Further support has been found for the suggestion that, 
bordering the S(q,q) phase for 0.25 < J2 < 0.7, S(q, 0), spiral order is unstable against a new 
form of order. For the case of AF J\ we find a picture for the nature of disordered phase next 
to the Q = (it, 7r) slightly different from the one proposed in [l] : mixed columnar-plaquette 
VBC order as proposed in [TU], and provide evidence for additional new phases at larger values 
of J3. In this paper we limit ourselves to the ground-state in zero magnetic field. The study of 
magnetization plateaux will be reported in a further publication. 

2. Ground-state phases 

Our main ED results for the spin-1/2 model are summarized in the phase diagrams Fig. Q] (FM 
Ji) and Fig. [2] (AF Ji) in which the boundaries of the spiral phases have been deduced from ED 
with TBC on iV = 32 clusters, and our conclusions on the nature of the ground-state include 
the results of an analysis of the spectra obtained by ED of up to = 36 spins. We note that 
these boundaries may differ from those of an infinite system. 

Classically, the J1-J2-J3 Heisenberg model on a square lattice has four magnetically ordered 
phases for J\ FM or AF [HJ [9]. These are collinear and spiral states at wave vector Q c / which 
extend over the regions labelled I-IV in Fig. Q] (FM J\ ) and Fig. [2] ( AF J\ ) : 

I - a FM phase with Q c ; = (0,0) for FM J%, a collinear phase with Q c ; = (tt, 7t) for AF J\, 
II - a collinear phase with Q c j = (it, 0) or (0, w), 
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Figure 2. (Color online). Groundstate phase diagram for J\ = 1. Symbols indicate 
the ground state found in exact diagonalization of a 32-site cluster, allowing for twisted 
boundary conditions : S(q,q) — spiral; SR S(q,ir) — sort-range order (see text); C(tt,tt), 
C(vr/2,vr), C(7r/2,vr/2) or C(vr,0) — collinear; MP-VBC — mixed columnar-plaquette VBC 
state; C-VBC — columnar VBC state; SG — spin-gapped. Dashed lines are the classical 
(S = oo) phase boundaries. 

III - a spiral phase with Q c ; = (q, 0) or (0, q) where q = cos 
Qd = (q, tt) or (it, q) with q = cos 

IV - a spiral phase with Q c ; = (q, q) or (q, —q) where q = cos 

for FM Ji and q G [vr/2,vr] for AF J\). 

These magnetic orders will be referred as FM, C(tt,tt), C(tt,0), S(q,Q) S(q,ir), and S(q,q) in 
this paper. 

At a classical level, all transitions are continuous since at the special point Ji = 0.5, J3 = 0, 
and on the boundary between III and IV, the classical ground state becomes degenerate with 
a family of spirals which interpolate continuously between the competing values of Q c ;, and 
appear as additional zero modes in linear spin wave theory (LSW) calculations. As a result of 
the high density of low- lying excitations, LSW predicts that the sublattice magnetization of the 
classically ordered states vanishes in the vicinity of all boundaries (except the boundary of the 
FM phase). Treated more accurately, quantum fluctuations may act to modify the wave vectors 
of spiral ground states O 0] , enhance the extension of AF collinear phases, or lead to entirely 
new forms of order. The resulting phase transitions will generally be 1 st order. 

2.1. Phase diagram for FM J\ 

The phase diagram of Fig. [TJ only differs from the one presented in [6] in the region 
0.25 < J2 < 0.7 below the S(q,q) phase which we now label as SR S(q,0) instead of S(q,0). 
There, as already noted in [6], the spectra of the iV = 32 cluster at the twist angle that 
minimize the energy do not exhibit a well defined tower of state caracteristic of spiral order. 
Moreover, in this region, the energy is a very flat function of the twist angles over a large region 
of twists which bridge between twists that allow to acomodate a S(q, 0) and a S(q, q) spiral. This 
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region is indeed close to the line boundary between classical regions III and IV which is a line 
of degenerate spirals interpolating between S(q,Q) and S(q,q) spirals. Spiral order appears thus 
quite unstable and another kind of ground-state occurs with short-range spin-spin correlations. 
In view of the spectra, this ground-state does not exhibit a p-nematic order that correspond to 
the selection of a preferential plane for the spins as found in jTTj and is probably spin-gapped. 



2.2. Phase diagram for AF J\ 

The phase diagram in the case of AF J\ is shown in Fig. [2j The extension of the spirals is 
there also much reduced, and the S(q, it) order is completely suppressed. One has in addition a 
region, labelled SR S(q,q), possibly spin-gapped, where the energy is minimized for TBC but 
the spectra does not show spiral order. This region is located between the VBC and S(q, q) 
where it has been proposed that a Z2 spin-liquid occurs [15] . The wave vector of the S(q,q) 
spiral is also shifted by quantum fluctuations [6] . The SG state is similar to the one found for FM 
J\. The collinear C(tt,Q) and C(ir,ir) phases clearly extends beyond their classical boundaries. 

The main difference is the appearence of a spin-gapped phase with likely VBC order. In 
region adjacent to the C(tt,tt) phase, this VBC phase exhibit, in adddition to columnar dimer- 
dimer correlations, strong plaquette correlations, largest around the line ( J2 + J3) j J\ = 1/2 and 
with decreasing J2, as previously pointed in pQ. However an analysis of the symmetry properties 
of the low lying singlets present in the spectra and their evolution with the size of the clusters 
indicates that an eight-fold degenerate mixed plaquette-columnar order, as proposed in [TS] and 
recently in [TO], or even possibly a more complex pattern, is much more likely than a purely 
plaquette VBC order. Moreover both the spectra and correlations seem to favor a (quasi) pure 
columnar (four-fold degenerate) VBC order for J2 > 0.5Ji. 

For 0.5 < J2/J1 < 0.75 and J3/J1 < 0.35, we found a region where the spectrum that 
suggests (four-fold degenerate) C(tt/2,it) collinear order. This state is one of the classical 
ground-states at J2/J1 = 0.5 for < J3/J1 < 0.25, degenerate with the S(ir/2,ir) spiral. But 
quantum fluctuations may not be sufficiently strong to stabilize this collinear state by an order 
by disorder mechanism, at variance with the FM Ji case, where one nicely sees the signature 
of the similar C(-7r/2,0) order. In the AF case, such a collinear state seems in competion with 
VBC order which could gain more from quantum fluctuations. The ground-state may be spin- 
gapped with possibly columnar VBC order, as suggested the correlations and the symmetry of 
the lowest singlets (which are the same as those of the singlets of the Anderson tower of the 
C(7r/2,7r) order) or even could be a spin-liquid. The nature of the ground state in this region 
and the regions labelled SR S(q,0), SR S(q,ir) or SR S(q,q) deserves further investigations. 
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